Apparatus and method for digital holographic table top display

ABSTRACT

Disclosed is an apparatus and method for digital holographic table top display. The digital holographic table top display apparatus includes: a camera array configured to capture a plurality of channel images in an omni-directional range from a table by using a plurality of cameras; a controller configured to detect an observer from the plurality of channel images and to track a position of pupils of the observer in at least one channel image from which the observer is detected; and a display configured to reproduce a digital holographic image in a three-dimensional (3D) space according to the tracked position of the pupils.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority from Korean Patent Application No.10-2014-0138429, filed on Oct. 14, 2014, in the Korean IntellectualProperty Office, the entire disclosure of which is incorporated hereinby reference for all purposes.

BACKGROUND

1. Field

The following description relates to a three dimensional displaytechnology.

2. Description of the Related Art

Three-dimensional (3D) images currently displayed through atwo-dimensional (2D) screen are different from real 3D images. There isa technical challenge in that motion parallax is not seamless as a realimage, such that when an image is viewed from a different angle, a usermay not see other sides viewed at the different angle. Further, when anobserver focuses on an object in a 3D space, an image should be providedin a manner that enables the observer to see the object without feelingfatigue.

However, the existing 2D screen based approach may not be suitable for a3D image reproduction method that may overcome the above challenge andsatisfy the need. Super multi-view images and the like may be used as asubstitute, but only the holographic images are optimal to provideperfect 3D images.

SUMMARY

Provided is an apparatus and method for digital holographic table topdisplay, in which a digital holographic image is provided at anydirection around 360 degrees according to the position of pupils of anobserver, thereby expanding a field of view.

In one general aspect, there is provided a digital holographic table topdisplay apparatus, including: a camera array configured to capture aplurality of channel images in an omni-directional range from a table byusing a plurality of cameras; a controller configured to detect anobserver from the plurality of channel images and to track a position ofpupils of the observer in at least one channel image from which theobserver is detected; and a display configured to reproduce a digitalholographic image in a three-dimensional (3D) space according to thetracked position of the pupils.

The camera array of the plurality of cameras may be arranged in a circletoward a center of the table to acquire images around 360 degrees.

The controller may include: a multi-grid image generator configured tocombine the channel images captured by the plurality of cameras togenerate one multi-grid image; an observer detector configured to detectat least one observer from the multi-grid image; a channel determinerconfigured to select a channel associated with a channel image fromwhich the observer is detected; a pupil tracker configured to track theposition of the pupils in the channel image associated with informationon the selected channel; and a coordinate calculator configured tocalculate 3D coordinates of the position of the pupils by using thetracked position of the pupils and the information on the selectedchannel.

The observer detector may extract location information on an additionalchannel area from the multi-grid image having the at least one channelimage from which the observer is detected, and transmits the extractedlocation information on the additional channel area along with thechannel information to the channel determiner.

The location information on the additional channel area may be locationinformation on a face area or location information on a face area and aneye area.

With respect to one multi-grid image having the at least one channelimage from which an observer is detected, the channel determiner maytransmit, to the pupil tracker, an original channel image captured bythe plurality of cameras, or an enlarged image from the at least onechannel image.

The coordinate calculator may calculate 3D coordinates of the positionof each of the pupils tracked in the channel images captured by twoadjacent stereo cameras, and may convert the calculated 3D coordinatesof the position of each of the pupils on the basis of a predeterminedreference point.

The predetermined reference point may be the center of the table.

The display may include an optical device configured to form a viewingwindow by controlling a direction of a beam to be directed to theposition of the pupils tracked by the controller, and to reproduce thedigital holographic image through the formed viewing window.

In another general aspect, there is provided a digital holographic tabletop display method, including: capturing a plurality of channel imagesin an omni-directional range from a table by using a camera array thatincludes a plurality of cameras; detecting an observer from theplurality of channel images and tracking a position of pupils of theobserver in at least one channel image from which the observer isdetected; and reproducing a digital holographic image in athree-dimensional (3D) space according to the tracked position of thepupils.

The tracking of the position of the pupils may include: generating onemulti-grid image by combining the channel images captured by theplurality of cameras; detecting at least one observer from themulti-grid image; selecting a channel associated with a channel imagefrom which the observer is detected; tracking the position of the pupilsin the channel image associated with information on the selectedchannel; and calculating 3D coordinates of the position of the pupils byusing the tracked position of the pupils and the information on theselected channel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a conceptual diagram illustrating a digital holographic tabletop display according to an embodiment, and FIG. 1B is a diagramillustrating a method of configuring a digital holographic table topdisplay according to an embodiment.

FIG. 2 is a block diagram illustrating a digital holographic table topdisplay apparatus according to an embodiment.

FIG. 3 is a diagram illustrating a face recognition method based on Haarfeatures.

FIG. 4 is a diagram illustrating an example of camera arrangement toexplain a process of generating 3D coordinates of pupils.

FIG. 5 is a flowchart illustrating a digital holographic image displaymethod according to an exemplary embodiment.

Throughout the drawings and the detailed description, unless otherwisedescribed, the same drawing reference numerals will be understood torefer to the same elements, features, and structures. The relative sizeand depiction of these elements may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following description is provided to assist the reader in gaining acomprehensive understanding of the methods, apparatuses, and/or systemsdescribed herein. Accordingly, various changes, modifications, andequivalents of the methods, apparatuses, and/or systems described hereinwill be suggested to those of ordinary skill in the art. Also,descriptions of well-known functions and constructions may be omittedfor increased clarity and conciseness. Terms used throughout thisspecification are defined in consideration of functions according toexemplary embodiments, and can be varied according to a purpose of auser or manager, or precedent and so on. Therefore, definitions of theterms should be made on the basis of the overall context.

FIG. 1A is a conceptual diagram illustrating a digital holographic tabletop display according to an embodiment, and FIG. 1B is a diagramillustrating a method of configuring a digital holographic table topdisplay according to an embodiment.

Referring to FIGS. 1A to 1B, in the digital holographic table topdisplay technology, a holographic image may be reproduced in a 3D spaceby light diffraction of a light source 10 using a spatial lightmodulator (SLM) 14. The holographic table top display outputs aholographic image 100 in a 3D space on a plane table 110 as illustratedin FIG. 1A, so that at least one observer may view a 3D image at anydirection around 360 degrees. The holographic 3D image 100 is an imagereproduced by reconstructing a holographic image using the SLM 14.

With respect to an optical structure of the holographic table topdisplay, a light diffraction angle of the light source 10 is controlledby using optical components, such as an parabolic mirror 12 and the likeas illustrated in FIG. 1B, to provide a floating image effect in theair. The light source 10 may be a coherent light source such as a laser,or a partially coherent light source such as a light emitting diode(LED).

The holographic table top display is based on the light source and theSLM 14 as in other holographic displays. For this reason, the pixel size(or pixel pitch) of the SLM 14 limits the viewing zone of an observer.In order to overcome the limitation, a method is required to adjustdirections of output light by using the light source 10 and the SLM 14according to the position of pupils of an observer. In the presentdisclosure, pupils of an observer may be accurately detected in a 3Dspace, and a digital holographic image may be reproduced in the 3D spaceaccording to the detected position of pupils, thereby overcoming alimited viewing zone.

FIG. 2 is a block diagram illustrating a digital holographic table topdisplay apparatus (hereinafter referred to as a “display apparatus”)according to an embodiment.

Referring to FIG. 2, the display apparatus 2 includes a camera array 20including a plurality of cameras 200-1, 200-2, . . . , and 200-n, acontroller 22, and a display 24.

The camera array 20 captures a plurality of channel images in anomni-directional range from a table by using the plurality of cameras200-1, 200-2, . . . , and 200-n. The camera array 20 may enable imagesto be captured around 360 degrees by using the plurality of cameras200-1, 200-2, . . . , and 200-n that are arranged in a circle toward thecenter of the table. For example, images may be captured by 16 camerasarranged in a ring shape around the table. Each of the cameras 200-1,200-2, . . . , and 200-n may include channel information. The cameras200-1, 200-2, . . . , and 200-n may be arranged at a regular interval orat a regular angle, or may be concentrated on a specific area dependingon operating environments.

The position of pupils may be detected by using both an omnidirectionalcamera that allows 360 degree observation and a camera array arrangedaround a table. However, the above method requires a separate camerainput channel, and a correlation between the omnidirectional camera andthe camera array is required to be calculated again. Further, distortionoccurring in an omnidirectional camera leads to an additionalcalculation to compensate for the distortion. In addition, since theomnidirectional camera is located at a different height from the cameraarray, face recognition capability of the omnidirectional camera isreduced. However, in the present disclosure, by using only the cameraarray without the omnidirectional camera, an exact position of pupilsmay be detected in a 3D space.

The controller 22 detects an observer from a plurality of channel imagescaptured by the cameras 200-1, 200-2, . . . , and 200-n and tracks aposition of pupils of the observer in at least one channel image fromwhich the observer is detected. In one exemplary embodiment, thecontroller 22 includes a multi-grid image generator 220, an observerdetector 222, a channel determiner 224, a pupil tracker 226, and acoordinate calculator 228.

The multi-grid image generator 220 generates one multi-grid image 2200by combining channel images captured by the camera array 20. Themulti-grid image generator 220 may scale down channel images captured bythe camera array 20 to generate thumbnail images, may combine thegenerated thumbnail images to generate one multi-grid image 2200 such as4-channel grid image and 16-channel grid image, and may transmit thegenerated multi-grid image to the observer detector 222.

The observer detector 222 detects at least one observer from themulti-grid image generated by the multi-grid image generator 220, andtransmits channel information regarding the image to the channeldeterminer 224. For example, in the case of using 16 channels asillustrated in FIG. 2, if an observer is detected from a channel area (apart or grid) #1 and a channel area #2 of the multi-grid image,information on channel areas #1 and #2 is transmitted to the channeldeterminer 224.

In one exemplary embodiment, the observer detector 222 extracts locationinformation on an additional channel area, from which an observer isdetected, and transmits the extracted location information to thechannel determiner 224 along with channel information. The locationinformation may be face position information. For example, the observerdetector 222 may transmit, to the channel determiner 224, face positioninformation, e.g., information on a location in a square area that is 80in width and 60 in length from starting points 100 and 120 of channel#1.

In another exemplary embodiment, the observer detector 222 may detectthe position of eyes in the case of a multi-grid image that includeschannel images, such as a quartered image, which has a specific size. Inthis case, the observer detector 222 transmits information on thedetected position of eyes along with channel information. Although FIG.2 illustrates an example of detecting an identical observer, differentobservers may also be detected.

The channel determiner 224 selects a channel associated with a channelarea (i.e., channel image) from which an observer is detected by theobserver detector 222. That is, the channel determiner 224 transmits, tothe pupil tracker 226, only the channel information associated with achannel area, from which an observer is detected, among imagestransmitted from the observer detector 222. Instead of transmittinginformation on all the channels, only the channel information associatedwith areas, from which an observer is detected, is transmitted, therebyimproving efficiency in tracking positions of pupils. The channeldeterminer 224 may transmit information on one or more channels to thepupil tracker 226. For example, information on at least two channels istransmitted per person to generate 3D coordinates of pupils based on astereo camera.

When selecting channels, the channel determiner 224 has a switchingfunction, which connects an input channel and output channels accordingto the channel information and a predetermined channel environment. Inthis case, the channel determiner 224 may transmit, to the pupil tracker226, an original high-resolution channel image captured by a camera, oran enlarged image from a channel image. In this case, the pupil tracker226 may track the position of pupils in the high-resolution channelimage transmitted from the channel determiner 224.

The pupil tracker 226 tracks the position of pupils by receiving channelimages from the channel determiner 224. In the case of receivingadditional information associated with location information on a facearea or an eye area, the position of pupils is tracked in detail usingthe location information on a specific area. The pupil tracker 226transmits the tracked position of the pupils to the coordinatecalculator 228 along with channel information. The channel informationmay be transmitted directly from the observer detector 222 or thechannel determiner 224 to the coordinate calculator 228.

In one exemplary embodiment, the pupil tracker 226 tracks the positionof pupils in the high-resolution channel image input from the channeldeterminer 224. In the case where there is location information on aface area calculated by the observer detector 222, a position of an eyearea is tracked based on the face area. By contrast, in the case wherethere is no location information on a face area, a face area is firstdetected in the same manner as in a method of detecting a face position,and an eye area is detected from the face area; however, in the casewhere the pupil tracker 226 receives location information on an eye areadetected by the observer detector 222, a detailed location of an eyearea is detected by using the received location information on an eyearea.

In another exemplary embodiment, in the case where there is locationinformation on a face area calculated by the observer detector 222 whenthe channel determiner 225 selects channels, only the face area of ahigh-resolution channel image may be transmitted to the pupil tracker226. In this manner, only the data on a face area may be transmitted,thereby enabling fast detection of location information on an eye area.

A detailed eye position, i.e., pupils, may be detected by a generalmethod used for detecting pupils or eyes. The position of pupils may bedetected by using characteristics indicating that pupils are round oroval and characteristics indicating that pupils look darker thansurrounding areas when captured by cameras. As an example of using shapecharacteristics of eyes, a circle detection algorithm that comparesaccumulated values of brightness differences between surroundingboundaries of the eyes may be indicated by the following Equation 1.

$\begin{matrix}{{\max\limits_{({r,x_{0},y_{0}})}{{{G_{\sigma}(r)}*\frac{\partial\;}{\partial r}{\oint_{r \times x_{0}y_{0}}{\frac{I\left( {x,y} \right)}{2\pi \; r}{s}}}}}},} & (1)\end{matrix}$

in which I(x, y) represents a pixel value at (x, y), (x₀, y₀) representsthe center of a circle, and r represents a radius. In Equation 1, byadding all the pixel values around the circumference of the circle thatis normalized to be 2πr by radius r from the center (x₀, y₀) of thecircle, a pupil area is determined to be an area having the biggestdifference between pixel values of an inner circumference and pixelvalues of an outer circumference, in which Gaussian function G(r) isperformed in a direction of radius r so as to remove noise whenextracting the position of pupils. In another example, a pupil area isdetermined by detecting the darkest area by using brightnessdifferences, and by detecting an area that is most similar to a circlein the darkest area. The above methods are merely illustrative, and thepresent disclosure is not limited thereto.

The coordinate calculator 228 calculates a 3D position of pupils byusing channel information of cameras and location information on thedetected pupils. The channel information of cameras may be transmittedfrom the pupil tracker 226, the observer detector 222, or the channeldeterminer 224. The coordinate calculator 228 may transmit the 3Dposition of pupils to the display 24.

The display 24 reproduces a digital holographic image in a 3D spaceaccording to the position of pupils tracked by the controller 22. Thedisplay 24 may include an optical device that forms a viewing window bycontrolling a beam direction to be directed to the pupil position, andreproduces a digital holographic image through the formed viewingwindow. The viewing window is a virtual window in an observer area, inwhich a reconstructed 3D image may be viewed.

FIG. 3 is a diagram illustrating a face recognition method based on Haarfeatures.

Referring to FIGS. 2 and 3, in one exemplary embodiment, a facerecognition method based on Haar features is used to recognize a face.As illustrated in FIG. 3, the method includes generating feature vectorsfor input images and inputting the feature vectors to a general faceclassifier to recognize a face.

In another example, if each channel image in a multi-grid image is of abig size, e.g., a four-channel or eight-channel image, an eye area mayalso be detected. For example, in the case of using a method based onHaar features, similarities may be compared in such a manner that Haarfeature vectors are collected again for an area that is recognized as aface and are compared with feature vectors of an eye classifier.However, the above face recognition methods are merely illustrative toassist in understanding the present disclosure, and any general methodfor face recognition may also be used.

FIG. 4 is a diagram illustrating an example of camera arrangement toexplain a process of generating 3D coordinates of pupils.

Referring to FIGS. 2 and 4, the coordinate calculator 228 may calculate3D coordinates of pupils by using channel information selected by thechannel determiner 224 and pupil position information calculated by thepupil tracker 226. The channel information may provide information onthe position of cameras that input channel images. In a stereo camerasetup, a distance between cameras and an observer and a position ofpupils may be calculated by using a disparity between pupils captured bytwo adjacent cameras.

There is a disparity in 3D positions of pupils tracked in channel imagescaptured by the two adjacent cameras. The coordinate calculator 228 mayobtain a specific 3D position by converting 3D coordinates of pupils onthe basis of a predetermined reference point. For example, in the casewhere observers and cameras are arranged as illustrated in FIG. 4, adisparity occurs between 3D coordinates of pupils captured by camera 1and 3D coordinates of pupils captured by camera 2. The coordinatecalculator 228 converts 3D coordinates of pupils on the basis of thecenter of a table. The calculation may be performed in the same manneras a general 3D transformation. Equation 2 represents coordinates(X_(c), Y_(c), Z_(c)) converted, on the basis of center points of atable, from 3D coordinates (X′, Y′, Z′) captured by the cameras.

The 3D coordinates calculated by the coordinate calculator 228 aretransmitted to the display 24. The display 24 may generate a digitalholographic image according to the calculated position of an observer'spupils.

FIG. 5 is a flowchart illustrating a digital holographic image displaymethod according to an exemplary embodiment.

Referring to FIGS. 2 and 5, a display device 2 acquires in 500 aplurality of channel images in an omni-directional range from a table byusing a camera array 20 that includes a plurality of cameras 200-1,200-2, . . . , and 200-n. By arranging the plurality of cameras 200-1,200-2, . . . , and 200-n included in the camera array 20 in a circletoward the center of a table, images may be captured around 360 degrees.

Subsequently, the display device 2 generates one multi-grid image bycombining channel images captured by the plurality of cameras 200-1,200-2, . . . , and 200-n. Then, at least one observer is detected fromthe multi-grid image in 510. In the detection of an observer in 510,location information on an additional channel area may be extracted fromat least one channel area, from which the observer is detected, in themulti-grid image. The location information on an additional channel areamay be a face area, or a face area and an eye area.

Then, the display device 2 selects channels regarding channel areas fromwhich an observer is detected, and the position of pupils is tracked inthe channel images in 520 associated with the selected channels.

Next, a 3D position of pupils is calculated by using locationinformation of the tracked pupils and the channel information, toreproduce a digital holographic image in a 3D space according to thecalculated 3D position of pupils in 530. When the 3D position iscalculated, specific 3D coordinates may be obtained by calculating 3Dcoordinates of pupils tracked in channel images captured by two adjacentstereo cameras, and by converting the calculated 3D coordinates ofpupils on the basis of predetermined reference information. Inreproducing a digital holographic image in 530, a viewing window isformed by controlling a beam direction to be directed to the calculated3D position of pupils, and a digital holographic image may be reproducedthrough the formed viewing window.

As described above, in the digital holographic table top display, pupilsof an observer may be detected accurately in a 3D space by using aplurality of cameras, and a digital holographic image may be reproducedin the 3D space according to the detected position of the pupils,thereby overcoming a limited field of view.

A number of examples have been described above. Nevertheless, it shouldbe understood that various modifications may be made. For example,suitable results may be achieved if the described techniques areperformed in a different order and/or if components in a describedsystem, architecture, device, or circuit are combined in a differentmanner and/or replaced or supplemented by other components or theirequivalents. Accordingly, other implementations are within the scope ofthe following claims. Further, the above-described examples are forillustrative explanation of the present invention, and thus, the presentinvention is not limited thereto.

What is claimed is:
 1. A digital holographic table top displayapparatus, comprising: a camera array configured to capture a pluralityof channel images in an omni-directional range from a table by using aplurality of cameras; a controller configured to detect an observer fromthe plurality of channel images and to track a position of pupils of theobserver in at least one channel image from which the observer isdetected; and a display configured to reproduce a digital holographicimage in a three-dimensional (3D) space according to the trackedposition of the pupils.
 2. The apparatus of claim 1, wherein the cameraarray of the plurality of cameras is arranged in a circle toward acenter of the table to acquire images around 360 degrees.
 3. Theapparatus of claim 1, wherein the controller comprises: a multi-gridimage generator configured to combine the channel images captured by theplurality of cameras to generate one multi-grid image; an observerdetector configured to detect at least one observer from the multi-gridimage; a channel determiner configured to select a channel associatedwith a channel image from which the observer is detected; a pupiltracker configured to track the position of the pupils in the channelimage associated with information on the selected channel; and acoordinate calculator configured to calculate 3D coordinates of theposition of the pupils by using the tracked position of the pupils andthe information on the selected channel.
 4. The apparatus of claim 3,wherein the observer detector extracts location information on anadditional channel area from the multi-grid image having the at leastone channel image from which the observer is detected, and transmits theextracted location information on the additional channel area along withthe channel information to the channel determiner.
 5. The apparatus ofclaim 4, wherein the location information on the additional channel areais location information on a face area or location information on a facearea and an eye area.
 6. The apparatus of claim 3, wherein with respectto one multi-grid image having the at least one channel image from whichan observer is detected, the channel determiner transmits, to the pupiltracker, an original channel image captured by the plurality of cameras,or an enlarged image from the at least one channel image.
 7. Theapparatus of claim 3, wherein the coordinate calculator calculates 3Dcoordinates of the position of each of the pupils tracked in the channelimages captured by two adjacent stereo cameras, and converts thecalculated 3D coordinates of the position of each of the pupils on thebasis of a predetermined reference point.
 8. The apparatus of claim 7,wherein the predetermined reference point is the center of the table. 9.The apparatus of claim 1, wherein the display comprises an opticaldevice configured to form a viewing window by controlling a direction ofa beam to be directed to the position of the pupils tracked by thecontroller, and to reproduce the digital holographic image through theformed viewing window.
 10. A digital holographic table top displaymethod, comprising: capturing a plurality of channel images in anomni-directional range from a table by using a camera array thatincludes a plurality of cameras; detecting an observer from theplurality of channel images and tracking a position of pupils of theobserver in at least one channel image from which the observer isdetected; and reproducing a digital holographic image in athree-dimensional (3D) space according to the tracked position of thepupils.
 11. The method of claim 10, wherein the capturing of theplurality of channel images comprises capturing images around 360degrees by using the camera array that includes the plurality of camerasarranged in a circle toward a center of the table.
 12. The method ofclaim 10, wherein the tracking of the position of the pupils comprises:generating one multi-grid image by combining the channel images capturedby the plurality of cameras; detecting at least one observer from themulti-grid image; selecting a channel associated with a channel imagefrom which the observer is detected; tracking the position of the pupilsin the channel image associated with information on the selectedchannel; and calculating 3D coordinates of the position of the pupils byusing the tracked position of the pupils and the information on theselected channel.
 13. The method of claim 12, wherein the detecting ofthe observer comprises: extracting location information on an additionalchannel area from the multi-grid image having the at least one channelimage from which the observer is detected; and transmitting theextracted location information on the additional channel area along withthe channel information.
 14. The method of claim 12, wherein theselecting of the channel comprises: with respect to one multi-grid imagehaving the at least one channel image from which the observer isdetected, transmitting an original channel image captured by theplurality of cameras, or an enlarged image from the at least one channelimage.
 15. The method of claim 12, wherein the calculating of the 3Dcoordinates comprises: calculating the 3D coordinates of the position ofeach of the pupils tracked in the channel images captured by twoadjacent stereo cameras; and converting the calculated 3D coordinates ofthe position of each of the pupils on the basis of a predeterminedreference point.
 16. The method of claim 12, wherein the reproducing ofthe digital holographic image comprises: forming a viewing window bycontrolling a direction of a beam to be directed to the position of thepupils tracked by the controller; and reproducing the digitalholographic image through the formed viewing window.